Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus, an image processing method, and a program capable of acquiring correct scope information and acquiring accurate depth information from an intraluminal image. In image processing apparatus (14) including a processor, the processor performs image acquisition processing of acquiring a time-series intraluminal image captured by a scope of an endoscope; scope information acquisition processing of acquiring scope information relating to a change of the scope; landmark recognition processing of recognizing a landmark in the intraluminal image; scope information correction processing of correcting the scope information using information relating to the landmark recognized in the landmark recognition processing; and depth information acquisition processing of acquiring depth information of the intraluminal image using the intraluminal image and the scope information corrected in the scope information correction processing.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of PCT InternationalApplication No. PCT/JP2022/010892 filed on Mar. 11, 2022 claimingpriority under 35 U.S.C § 119(a) to Japanese Patent Application No.2021-047136 filed on Mar. 22, 2021. Each of the above applications ishereby expressly incorporated by reference, in its entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and a program, and more particularly relates to animage processing apparatus, an image processing method, and a program ofacquiring depth information of an intraluminal image.

2. Description of the Related Art

In observation performed using an endoscope system (endoscopeapparatus), a technique of displaying the position of an endoscopicscope of the endoscope system, the shape of a lumen, and the position ofa lesion in association with each other is known. This technique caneffectively assist a user in comprehensively observing a lumen (forexample, a large intestine) that is an observation target. Togeometrically recognize the current position of the endoscopic scope,the shape of the lumen, and the position of the lesion, it is necessaryto accurately estimate the depth from a camera included in a tip part ofthe scope to a target object.

WO2007/139187A proposes a technique of acquiring distance information(depth information) based on brightness information of an endoscopicimage and constructing a three-dimensional image. Also, WO2007/139187Adescribes a technique of acquiring a change amount in an axial directionand a change amount in a circumferential direction of the endoscopicscope by a motion detection sensor, and correcting a developed imagebased on the acquired change amounts.

SUMMARY OF THE INVENTION

In the above-described technique, to acquire correct depth information,it is necessary to correctly acquire scope information relating to achange of the endoscopic scope (for example, an insertion length of thescope into a lumen, and a bending angle and a rotation amount of thescope in the lumen).

However, since the lumen of the observation target of the endoscopesystem is a non-rigid body, scope information of an actual measurementvalue acquired by a sensor or the like described in WO2007/139187A isnot correct in some cases. That is, the scope information acquired bythe sensor or the like may be different from the actual relative changeamount of the scope in the lumen. As described above, when the scopeinformation cannot be correctly acquired, the depth information acquiredusing the scope information is also incorrect.

The present invention is made in view of the situation, and an object ofthe invention is to provide an image processing apparatus, an imageprocessing method, and a program capable of acquiring correct scopeinformation and acquiring accurate depth information from anintraluminal image.

An image processing apparatus according to an aspect of the presentinvention to attain the above-described object is an image processingapparatus including a processor. The processor performs imageacquisition processing of acquiring a time-series intraluminal imagecaptured by a scope of an endoscope; scope information acquisitionprocessing of acquiring scope information relating to a change of thescope; landmark recognition processing of recognizing a landmark in theintraluminal image; scope information correction processing ofcorrecting the scope information using information relating to thelandmark recognized in the landmark recognition processing; and depthinformation acquisition processing of acquiring depth information of theintraluminal image using the intraluminal image and the scopeinformation corrected in the scope information correction processing.

With this aspect, the landmark in the intraluminal image is recognized,and the scope information is corrected using the information relating tothe recognized landmark. Accordingly, accurate depth information of theintraluminal image can be acquired based on the accurate scopeinformation.

Preferably, with reference to a position of the scope at a time T, thescope information acquisition processing acquires a change amount of aninsertion length of the scope and change amounts relating to bending androtation of the scope at a time T+α.

Preferably, the scope information acquisition processing acquiresinformation relating to an insertion length of the scope and bending anda rotation of the scope from an operation of an operation section of thescope.

Preferably, the landmark recognition processing recognizes a temporalchange of a correspondence point of the landmark, and the scopeinformation correction processing corrects the scope information usingthe temporal change of the correspondence point.

Preferably, the landmark recognition processing outputs recognitionreliability of the recognized landmark, and the scope informationcorrection processing determines whether to execute the correction ofthe scope information based on the recognition reliability, and performsthe correction based on a result of the determination.

With this aspect, the recognition reliability of the landmark is output,and it is determined whether to execute the correction of the scopeinformation based on the recognition reliability. Accordingly, accuratecorrection can be executed and accurate distance information can beacquired.

Preferably, the scope information correction processing outputs acorrection value obtained from the information relating to the landmark,determines whether to execute the correction based on the correctionvalue, and performs the correction based on a result of thedetermination.

With this aspect, the correction value is output from the informationrelating to the landmark, and it is determined whether to execute thecorrection based on the output correction value. Accordingly, accuratecorrection can be executed and accurate distance information can beacquired.

Preferably, the processor performs display control processing ofdisplaying geometric information relating to a lumen on a display unitbased on the depth information acquired in the depth informationacquisition processing.

With this aspect, since the geometric information relating to the lumenis displayed on the display unit based on the acquired depthinformation, correct geometric information relating to the scope can beprovided to the user.

Preferably, the geometric information is at least one of a shape of thelumen, a position of a lesion, a position of the scope, or a position ofa treatment tool.

An image processing method according to another aspect of the presentinvention is an image processing method using an image processingapparatus including a processor. The method, performed by the processor,includes an image acquisition step of acquiring a time-seriesintraluminal image captured by a scope of an endoscope; a scopeinformation acquisition step of acquiring scope information relating toa change of the scope; a landmark recognition step of recognizing alandmark in the intraluminal image; a scope information correction stepof correcting the scope information using information relating to thelandmark recognized in the landmark recognition step; and a depthinformation acquisition step of acquiring depth information of theintraluminal image using the intraluminal image and the scopeinformation corrected in the scope information correction step.

A program according to still another aspect of the present invention isa program causing an image processing method to be executed using animage processing apparatus including a processor. The program causingthe processor to execute an image acquisition step of acquiring atime-series intraluminal image captured by a scope of an endoscope; ascope information acquisition step of acquiring scope informationrelating to a change of the scope; a landmark recognition step ofrecognizing a landmark in the intraluminal image; a scope informationcorrection step of correcting the scope information using informationrelating to the landmark recognized in the landmark recognition step;and a depth information acquisition step of acquiring depth informationof the intraluminal image using the intraluminal image and the scopeinformation corrected in the scope information correction step.

According to the present invention, since the landmark in theintraluminal image is recognized, and the scope information is correctedusing the information relating to the recognized landmark, accuratedepth information of the intraluminal image can be acquired based onaccurate scope information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view schematically illustrating an endoscopic image and adepth image obtained from the endoscopic image;

FIG. 2 is a view illustrating geometric information relating to a largeintestine and a corresponding endoscopic image;

FIG. 3 is a diagram explaining acquisition of depth information from anendoscopic image and scope information;

FIG. 4 is a schematic diagram illustrating the entire configuration ofan endoscope system including an image processing apparatus;

FIG. 5 is a block diagram illustrating an embodiment of the imageprocessing apparatus;

FIG. 6 is a diagram explaining acquisition of scope information;

FIG. 7 is a diagram explaining acquisition of scope information;

FIG. 8 is a view explaining an example of a landmark;

FIG. 9 is a diagram explaining information relating to a landmark;

FIG. 10 is a diagram explaining acquisition of a depth image fromcorrected scope information and an intraluminal image;

FIG. 11 is a view illustrating an example of geometric information of alumen displayed on a display unit;

FIG. 12 is a diagram explaining a flow of acquisition of depthinformation;

FIG. 13 is a diagram explaining a flow of acquisition of depthinformation;

FIG. 14 is a flowchart presenting an image processing method; and

FIG. 15 is a flowchart presenting an image processing method.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of an image processing apparatus, animage processing method, and a program according to the presentinvention will be described with reference to the accompanying drawings.

Acquisition of Depth Information from Endoscopic Image

First, acquisition of depth information from an endoscopic image will bedescribed.

FIG. 1 is a view schematically illustrating an endoscopic image and adepth image obtained from the endoscopic image.

FIG. 1 illustrates an intraluminal image P that is an example of anendoscopic image acquired by an endoscope system 9 (FIG. 4 ). Theintraluminal image P is, for example, an image obtained byimage-capturing the inside of a large intestine. In the intraluminalimage P, a fold 101 in the large intestine is presented, and a tubularshape continues in an arrow direction. A depth image I is an imagehaving depth information corresponding to the intraluminal image P. Thedepth image I has information relating to a depth (distance) from acamera (for example, an imaging element 28 (FIG. 4 )). In the depthimage I, depth information is presented in a heat map form. Note thatthe depth image I is presented in a simplified manner, and specifically,seven regions having mutually different items of depth information arepresented. Note that, in the depth image I, the depth information may beactually presented in a heat map form with finer regions, and forexample, a different item of depth information may be presented everypixel. Also, in this example, as an example of an intraluminal image, acase where the large intestine is observed by the endoscope system 9will be described, but the example of the intraluminal image is notlimited thereto. The intraluminal image may be an image obtained byimage-capturing another luminal organ.

Normally, the above-described depth image I having the depth informationis acquired using images at a plurality of viewpoints whose relativepositional relationship is known, such as a stereo camera. However,since the endoscope system 9 includes a monocular camera, when the depthinformation is acquired, it is necessary to acquire the depthinformation based on an endoscopic image acquired by the monocularcamera.

For example, a document (Daniel Freedman et al., “Detecting DeficientCoverage in Colonoscopies”, CVPR2020,https://arxiv.org/pdf/2001.08589.pdf) describes a technique of acquiringa depth image having depth information from an endoscopic image acquiredby a monocular camera using a recognizer constituted of a convolutionalneural network (CNN).

When the depth information is acquired using only the endoscopic imagecaptured by the above-described monocular camera, a relative depth isestimated while the movement amount of an endoscopic scope 10 (see FIG.4 ) between adjacent frames is estimated. In this case, since the organis a non-rigid body, the lumen shape changes every frame, and an errormay occur in the depth information.

FIG. 2 is a view illustrating geometric information relating to a largeintestine and a corresponding endoscopic image.

Geometric information 500 relating to the large intestine indicates thecurrent position of an insertion section 20 of the endoscopic scope 10.Also, an endoscopic image acquired at the position of the endoscopicscope 10 indicated in the geometric information 500 of the largeintestine is presented.

As illustrated in FIG. 2(A), an intraluminal image P1 is acquired at theposition of the insertion section 20 indicated in the geometricinformation 500. As illustrated in FIG. 2(B), when the endoscopic scope10 is translated, bent, and rotated from the position indicated in FIG.2(A), an intraluminal image P2 is acquired. The viewpoint and the lumenshape change between the intraluminal image P1 and the intraluminalimage P2 due to the movement of the endoscopic scope 10 and theinfluence of the observation target (large intestine) that is anon-rigid body. In such a case, when depth information is acquired usingonly an intraluminal image captured by the above-described monocularcamera, an error may occur in the depth information.

As described above, since the observation target is a non-rigid body, inthe case where the depth information is acquired from the intraluminalimage, accuracy may be decreased when the depth information is acquiredusing only the intraluminal image.

Acquisition of Depth Information from Endoscopic Image and ScopeInformation

To suppress the above-described decrease in the accuracy of the depthinformation, it is considered that an actual measurement value of scopeinformation is acquired, and depth information is acquired from thescope information and an endoscopic image.

FIG. 3 is a diagram explaining acquisition of depth information from anendoscopic image and scope information.

As illustrated in FIG. 3 , scope information S and an intraluminal imageP are input to a depth information acquisition unit 45 (see FIG. 5 ),and a depth image I having depth information is output. Here, the scopeinformation S is information indicating a change of the endoscopic scope10, and is an actual measurement value. For example, the scopeinformation S is an insertion length of the endoscopic scope 10, and abending angle and a rotation amount of the endoscopic scope 10. Also,the depth information acquisition unit 45 is a learned model constitutedof a CNN, and learning is performed to output a depth image I inresponse to input of scope information S and an intraluminal image P. Asdescribed above, by inputting the scope information S of the actualmeasurement value to the depth information acquisition unit 45,calibration of the movement change amount of the endoscopic scope 10(specifically, the position of the imaging element 28) can be performed,and more correct depth information can be acquired.

Correction of Scope Information

Next, correction of the scope information S of the actual measurementvalue described above will be described.

The scope information S is an actual measurement value, and basically,an error does not occur with respect to the change amount of theendoscopic scope 10 in the actual lumen. However, since the observationtarget is a non-rigid body organ, the scope information S obtained as anactual measurement value and the relative change amount by which thescope has changed in the lumen do not coincide with each other in somecases. In such a case, the accuracy of the depth image I output from thedepth information acquisition unit 45 based on the scope information Sand the intraluminal image P may be low.

Here, by correcting the scope information S that is an actualmeasurement value using information relating to a landmark in theendoscopic image, information closer to the relative change amount ofthe endoscopic scope 10 in the lumen can be acquired. Thus, the presentinvention proposes a method of correcting scope information S of anactual measurement value using a landmark in an endoscopic image, andacquiring more correct depth information based on the corrected scopeinformation T.

First Embodiment Entire Configuration of Endoscope System IncludingImage Processing Apparatus

FIG. 4 is a schematic diagram illustrating the entire configuration ofan endoscope system including an image processing apparatus.

As illustrated in FIG. 4 , an endoscope system 9 includes an endoscopicscope 10 that is an electronic endoscope, a light source device 11, anendoscope processor device 12, a display device 13, an image processingapparatus 14, an operating unit 15, and a display unit 16.

The endoscopic scope 10 captures a time-series endoscopic imageincluding a subject image, and is, for example, a lower or uppergastrointestinal tract scope. The endoscopic scope 10 has an insertionsection 20 that is inserted into a subject (for example, stomach) andhas a distal end and a proximal end, a hand operation section 21 that isconnected to the proximal end side of the insertion section 20 and thatis gripped by a doctor, who is an operator, to perform variousoperations, and a universal cord 22 that is connected to the handoperation section 21. Also, a rotation scale 24 is provided at theendoscopic scope 10. The user can obtain the rotation amount in acircumferential direction of the endoscopic scope 10 by reading therotation scale 24. Here, the circumferential direction is acircumferential direction of a circle centered on the axis of theendoscopic scope 10.

The insertion section 20 is formed in an elongated shape with a smalldiameter as a whole. The insertion section 20 is constituted bycontinuously providing, in order from the proximal end side to thedistal end side thereof, a soft part 25 having flexibility, a bendingpart 26 bendable by an operation of the hand operation section 21, and atip part 27 in which an imaging optical system (objective lens) (notillustrated), an imaging element 28, and the like, are incorporated.Note that, a length scale 34 indicating an insertion length (push-inamount) of the insertion section 20 is provided at the insertion section20. The user can obtain the insertion length of the insertion section 20by reading the length scale 34.

The imaging element 28 is a complementary metal oxide semiconductor(CMOS) type imaging element or a charge coupled device (CCD) typeimaging element. Image light of a part to be observed is incident on animaging surface of the imaging element 28 via an observation window (notillustrated) opened in a distal end surface of the tip part 27 and theobjective lens (not illustrated) disposed behind the observation window.The imaging element 28 captures the image light of the part to beobserved incident on the imaging surface thereof (converts the imagelight into an electric signal), and outputs an imaging signal. That is,endoscopic images are sequentially captured by the imaging element 28.Note that the endoscopic images are acquired as a moving image 38 and astill image 39 (described later).

The hand operation section 21 is provided with various operation membersthat are operated by the doctor (user). Specifically, the hand operationsection 21 is provided with two types of bending operation knobs 29 usedfor bending operations of the bending part 26, an air/water supplybutton 30 for an air/water supply operation, and a suction button 31 fora suction operation. Also, the hand operation section 21 is providedwith a still image capturing instruction portion 32 for giving aninstruction to capture a still image 39 of the part to be observed, anda treatment tool lead-in port 33 for inserting a treatment tool (notillustrated) into a treatment tool insertion passage (not illustrated)inserted through the insertion section 20.

The universal cord 22 is a connection cord for connecting the endoscopicscope 10 to the light source device 11. The universal cord 22incorporates a light guide 35, a signal cable 36, and a fluid tube (notillustrated) that are inserted through the insertion section 20. Also, aconnector 37 a that is connected to the light source device 11 and aconnector 37 b that is branched from the connector 37 a and connected tothe endoscope processor device 12 are provided at an end portion of theuniversal cord 22.

By connecting the connector 37 a to the light source device 11, thelight guide 35 and the fluid tube (not illustrated) are inserted intothe light source device 11. Accordingly, necessary illumination light,water, and gas are supplied from the light source device 11 to theendoscopic scope 10 via the light guide 35 and the fluid tube (notillustrated). As a result, illumination light is emitted from anillumination window (not illustrated) in the distal end surface of thetip part 27 toward the part to be observed. Also, in accordance with apressing operation of the above-described air/water supply button 30, agas or water is ejected from an air/water supply nozzle (notillustrated) in the distal end surface of the tip part 27 toward theobservation window (not illustrated) in the distal end surface.

By connecting the connector 37 b to the endoscope processor device 12,the signal cable 36 and the endoscope processor device 12 areelectrically connected to each other. Accordingly, via the signal cable36, an imaging signal of the part to be observed is output from theimaging element 28 of the endoscopic scope 10 to the endoscope processordevice 12, and a control signal is output from the endoscope processordevice 12 to the endoscopic scope 10.

The light source device 11 supplies illumination light to the lightguide 35 of the endoscopic scope 10 via the connector 37 a. As theillumination light, light in various wavelength ranges according to theobservation purpose, such as white light (light in a white wavelengthrange or light in a plurality of wavelength ranges), light in one or aplurality of specific wavelength ranges, or a combination thereof, isselected.

The endoscope processor device 12 controls the operation of theendoscopic scope 10 via the connector 37 b and the signal cable 36.Also, the endoscope processor device 12 generates an image (alsoreferred to as a “moving image 38”) consisting of time-series frameimages 38 a including a subject image based on imaging signals acquiredfrom the imaging element 28 of the endoscopic scope 10 via the connector37 b and the signal cable 36. Further, when the still image capturinginstruction portion 32 is operated at the hand operation section 21 ofthe endoscopic scope 10, the endoscope processor device 12 sets oneframe image 38 a in the moving image 38 as a still image 39corresponding to the timing of the capturing instruction in parallelwith generation of the moving image 38.

The moving image 38 and the still image 39 are endoscopic imagesobtained by image-capturing the inside of a subject, that is, the insideof a living body. Further, when the moving image 38 and the still image39 are images obtained with light (special light) in the above-describedspecific wavelength range, both are special light images. Then, theendoscope processor device 12 outputs the generated moving image 38 andstill image 39 to the display device 13 and the image processingapparatus 14.

The endoscope processor device 12 may generate (acquire) a special lightimage having information of the above-described specific wavelengthrange based on a normal light image obtained with the above-describedwhite light. In this case, the endoscope processor device 12 functionsas a special light image acquisition unit. The endoscope processordevice 12 obtains a signal in the specific wavelength range byperforming calculation based on color information of red, green, andblue (RGB), or cyan, magenta, and yellow (CMY) included in the normallight image.

Alternatively, for example, the endoscope processor device 12 maygenerate a feature amount image such as a known oxygen saturation imagebased on at least one of a normal light image obtained with theabove-described white light or a special light image obtained with theabove-described light (special light) in the specific wavelength range.In this case, the endoscope processor device 12 functions as a featureamount image generation unit. Note that any of the moving image 38 orthe still image 39 including the in-vivo image, the normal light image,the special light image, and the feature amount image described above isan endoscopic image obtained by image-capturing a human body or byimaging a measurement result for the purpose of diagnosis or inspectionusing an image.

The display device 13 is connected to the endoscope processor device 12,and functions as the display unit 16 that displays the moving image 38and the still image 39 input from the endoscope processor device 12. Thedoctor (user) performs an advancing/retracting operation or the like ofthe insertion section 20 while checking the moving image 38 displayed onthe display device 13, when finding a lesion or the like in the part tobe observed, operates the still image capturing instruction portion 32to execute still image capturing of the part to be observed, andperforms a treatment such as diagnosis or biopsy. Note that the movingimage 38 and the still image 39 are similarly displayed on the displayunit 16 connected to the image processing apparatus 14 (describedlater). Also, when the moving image 38 and the still image 39 aredisplayed on the display unit 16, notification display (described later)is also performed together. Thus, the user preferably performs diagnosisor the like while viewing the display on the display unit 16.

Image Processing Apparatus

FIG. 5 is a block diagram illustrating an embodiment of the imageprocessing apparatus 14. The image processing apparatus 14 sequentiallyacquires time-series endoscopic images, and displays the endoscopicimages and geometric information relating to the lumen on the displayunit 16. The image processing apparatus 14 is constituted of, forexample, a computer. The operating unit 15 includes buttons provided atthe hand operation section 21 of the endoscopic scope 10 in addition toa keyboard, a mouse, and the like, connected to the computer in a wiredor wireless manner, and various monitors such as a liquid crystalmonitor connectable to the computer are used as the display unit 16.

The image processing apparatus 14 is composed of an image acquisitionunit 40, a central processing unit (CPU) 41, a scope informationacquisition unit 42, a landmark recognition unit 43, a scope informationcorrection unit 44, a depth information acquisition unit 45, a displaycontrol unit 46, a voice control unit 47, and a memory 48. Theprocessing of each unit is implemented by one or more processors. Here,the processor may be constituted of the CPU 41, or may be constituted ofone or more CPUs (not illustrated).

The CPU 41 operates based on various programs including an operationsystem and an endoscopic image processing program stored in the memory48, generally controls the image acquisition unit 40, the scopeinformation acquisition unit 42, the landmark recognition unit 43, thescope information correction unit 44, the depth information acquisitionunit 45, the display control unit 46, and the voice control unit 47, andfunctions as a portion of each of these units.

The image acquisition unit 40 performs image acquisition processing tosequentially acquire time-series endoscopic images. The imageacquisition unit 40 acquires time-series endoscopic images including asubject image from the endoscope processor device 12 (FIG. 4 ) using animage input/output interface (not illustrated) connected to theendoscope processor device 12 in a wired or wireless manner. In thisexample, the moving image 38 captured by the endoscopic scope 10 isacquired. Also, when the above-described still image 39 is capturedwhile the moving image 38 is being captured by the endoscopic scope 10,the image acquisition unit 40 acquires the moving image 38 and the stillimage 39 from the endoscope processor device 12. In this example, anintraluminal image P (FIG. 1 ) obtained by image-capturing a largeintestine will be described as an example of an endoscopic image.

The memory 48 includes a flash memory, a read-only memory (ROM), arandom access memory (RAM), a hard disk device, and the like. The flashmemory, the ROM, and the hard disk device are nonvolatile memories thatstore the operation system, various programs such as the endoscopicimage processing program, the captured still image 39, and the like.Also, the RAM is a volatile memory that can read/write data at highspeed and that functions as an area for temporarily storing the variousprograms stored in the nonvolatile memories and as a work area for theCPU 41.

The scope information acquisition unit 42 performs scope informationacquisition processing to acquire scope information relating to a changeof the endoscopic scope 10. The scope information is informationindicating an operation of the insertion section 20 of the endoscopicscope 10. Specifically, the scope information includes an insertionlength indicating the length by which the insertion section 20 of theendoscopic scope 10 is pushed into a lumen, a bending angle indicatingthe bending of the bending part 26, and a rotation amount indicating therotation of the endoscopic scope 10 in the circumferential direction.The scope information S can be acquired by actual measurement, and thescope information acquisition unit 42 can acquire the scope informationS by various methods. For example, when the scope informationacquisition unit 42 acquires the insertion length, the insertion lengthmay be acquired by image-capturing the length scale 34 provided at theinsertion section 20 by a camera, or the insertion length may beacquired by a sensor (not illustrated) provided together with the lengthscale 34. Also, for example, when the scope information acquisition unit42 acquires the bending angle, the bending angle may be acquired basedon the rotation amount of the bending operation knob 29, or the bendingangle may be acquired by a sensor (not illustrated) provided at thebending part 26. Further, for example, when the scope informationacquisition unit 42 acquires the rotation amount, the rotation scale 24provided at the endoscopic scope 10 may be image-captured by a camera,and the read rotation amount may be acquired, or the rotation amount inthe circumferential direction of the endoscopic scope 10 may be acquiredby a gyro sensor (not illustrated) incorporated in the hand operationsection 21.

FIGS. 6 and 7 are views explaining acquisition of scope information S.FIG. 6 is a diagram explaining acquisition of the insertion length inthe scope information S, and FIG. 7 is a diagram explaining the bendingangle and the rotation amount in the scope information S.

As illustrated in FIG. 6 , at a time T, the insertion section 20 isinserted into a lumen by a length a. At a time T+α, the insertionsection 20 is inserted into the lumen by a length a+b. The scopeinformation acquisition unit 42 acquires, as scope information S, achange amount of the insertion length at the time T+a with reference tothe insertion length of the insertion section 20 at the time T. That is,the scope information acquisition unit 42 acquires a length b as achange amount of the insertion length as scope information S.

As also illustrated in FIG. 7 , at a time T, the distal end of theinsertion section 20 has a bending angle of 0°, and a rotation amount inthe circumferential direction of 0. At a time T+α, the distal end of theinsertion section 20 has a bending angle c and a rotation amount d inthe circumferential direction. The scope information acquisition unit 42acquires, as scope information S, change amounts of the bending angleand the rotation amount at the time T+a with reference to the bendingangle and the rotation amount at the time T. In this case, the scopeinformation acquisition unit 42 acquires c as a change amount of thebending angle and d as a change amount of the rotation amount in thecircumferential direction as scope information S.

The landmark recognition unit 43 (FIG. 5 ) performs landmark recognitionprocessing to recognize a landmark in an endoscopic image. Here, thelandmark is a portion serving as a mark in the endoscopic image, and bytracking the landmark in time series, an operation (change amount) ofthe endoscopic scope 10 can be recognized. Specific examples of thelandmark include a fold of the large intestine, the duodenum, or thelike; a lesion such as a polyp; a start point, an end point, or anintermediate point of an organ (in the case of the large intestine, asplenic flexure, a hepatic flexure, an ileocecal portion, or the like);and the like. The landmark recognition unit 43 can recognize a landmarkin an endoscopic image by various methods. For example, the landmarkrecognition unit 43 is constituted of a recognizer (learned model)constituted of a CNN and has undergone machine learning, and recognizesa landmark from an input endoscopic image.

FIG. 8 is a view explaining an example of a landmark recognized by thelandmark recognition unit 43.

The landmark recognition unit 43 recognizes a landmark L that is alesion part in an intraluminal image P1. When the landmark recognitionunit 43 is constituted of the recognizer, a score relating to therecognition of the landmark L may be output. This score is used asrecognition reliability which will be described in a second embodiment.

The scope information correction unit 44 (FIG. 5 ) performs scopeinformation correction processing to correct the scope information Susing information relating to the landmark recognized by the landmarkrecognition unit 43. Here, the information relating to the landmark is,specifically, information relating to a temporal change in the positionof the landmark recognized in each of intraluminal images P continuousin time series. The scope information correction unit 44 can correct thescope information S using the information relating to the landmarkaccording to various aspects. For example, the scope informationcorrection unit 44 replaces the change amount of the endoscopic scope 10obtained based on the information relating to the landmark with thescope information acquired by the scope information acquisition unit 42to acquire corrected scope information T.

FIG. 9 is a diagram explaining information relating to a landmarkacquired by the scope information correction unit 44.

As described below, the scope information correction unit 44 tracks alandmark in time series, uses depth information corresponding to thelandmark as information relating to the landmark, and acquires a changeamount of the endoscopic scope 10 in a lumen.

First, the landmark recognition unit 43 recognizes a landmark L in anintraluminal image P1 at a time T. The landmark recognition unit 43 alsorecognizes the landmark L (a correspondence point of the landmark L) ina depth image Il corresponding to the intraluminal image P1. Then, thescope information correction unit 44 acquires depth information of thelandmark L at the time T.

At a time T+α, the landmark recognition unit 43 recognizes the landmarkL recognized at the time T also in an intraluminal image P2. Thelandmark recognition unit 43 also recognizes the landmark L (acorrespondence point of the landmark L) in a depth image 12corresponding to the intraluminal image P2. Then, the scope informationcorrection unit 44 acquires depth information of the landmark L at thetime T+α. Thereafter, the scope information correction unit 44 acquiresa change amount X of the insertion length of the endoscopic scope 10,acquires a change amount Y of the bending angle of the scope, andacquires a change amount Z of the rotation amount in the circumferentialdirection of the scope based on a temporal change of the depthinformation of the landmark L (the correspondence point of the landmarkL) from the time T to the time T+α. In this example, the change amountsX, Y, and Z are acquired based on the temporal change in the position ofthe landmark L between the times T and T+α; however, the change amountsX, Y, and Z may be acquired based on the temporal change in the positionof the landmark L among three or more times.

Then, the scope information correction unit 44 corrects the scopeinformation using the change amount of the scope acquired based on thelandmark L. For example, the scope information correction unit 44replaces the scope information S acquired by the scope informationacquisition unit 42 with the change amount of the endoscopic scope 10acquired based on the landmark L. Specifically, the scope informationcorrection unit 44 corrects the change amount b of the insertion lengthacquired by the scope information acquisition unit 42 to the changeamount X of the insertion length based on the landmark. Also, the scopeinformation correction unit 44 corrects the change amount c of thebending angle acquired by the scope information acquisition unit 42 tothe change amount Y of the bending angle based on the landmark. Further,the scope information correction unit 44 corrects the change amount d inthe circumferential direction acquired by the scope informationacquisition unit 42 to the change amount Z in the circumferentialdirection based on the landmark.

In the above-described example, the change amount of the endoscopicscope 10 is acquired using the landmark in the depth image; however, thepresent invention is not limited thereto. For example, the movement ofthe landmark in the intraluminal image P and the change amount of theendoscopic scope 10 may be estimated through machine learning or thelike. A recognizer that has undergone machine learning on the movement(change amount) of the landmark in the intraluminal image P with respectto a plurality of patterns of change amounts (translation, rotation, andbending) of the endoscopic scope 10 in advance is prepared. Then, thetranslation amount of the endoscopic scope 10 is calculated andestimated from the change amount of the landmark in the intraluminalimage P from the time T to the time T+a by the recognizer.

The depth information acquisition unit 45 (FIG. 5 ) performs depthinformation acquisition processing to acquire depth information of theendoscopic image based on the endoscopic image and the scope informationT corrected by the scope information correction unit 44. The depthinformation acquisition unit 45 is a learned model that is constitutedof a CNN and has undergone machine learning. When the corrected scopeinformation T (or the scope information S) and the time-seriesintraluminal images are input, the depth information acquisition unit 45outputs a depth image having depth information.

FIG. 10 is a diagram explaining acquisition of a depth image I fromcorrected scope information T and an intraluminal image P.

As illustrated in FIG. 10 , the scope information T corrected by thescope information correction unit 44 and the intraluminal image P areinput to the depth information acquisition unit 45. The scopeinformation T corrected by the scope information correction unit 44presents the change amount of the scope in the lumen more correctly.Accordingly, the depth information acquisition unit 45 can output adepth image I having more correct depth information.

The display control unit 46 (FIG. 5 ) generates image data for displaybased on the endoscopic image (moving image 38) acquired by the imageacquisition unit 40, and outputs the image data for display to thedisplay unit 16. Also, the display control unit 46 generates geometricinformation relating to the lumen and outputs the geometric informationto the display unit 16. The voice control unit 47 (FIG. 5 ) controlsvoice output by a speaker 17. For example, the voice control unit 47controls the speaker 17 to output a notification sound to the user.

FIG. 11 is a view illustrating an example of an intraluminal image P andgeometric information F of a lumen displayed on the display unit 16.

An intraluminal image P captured by the endoscope system 9 is displayedin a main region of the display unit 16. Geometric information F of thelumen is displayed in a sub-region of the display unit 16. The geometricinformation F of the lumen is generated by the display control unit 46and output to the display unit 16. The geometric information F isgenerated based on the corrected scope information T and based on theaccurate depth information acquired by the depth information acquisitionunit 45. The geometric information F indicates the shape of the lumen(the shape of the large intestine) that is the observation target andthe current position of the endoscopic scope 10. Also, the geometricinformation F may indicate the position of a lesion, the position of atreatment tool, and the like. As described above, since the depthinformation acquisition unit 45 acquires accurate depth information, thegeometric information F using the depth information can correctlypresent position information or the like.

Image Processing Method and Program

Next, an image processing method using the image processing apparatus 14and a program for causing the image processing apparatus 14 to executethe image processing method will be described.

FIGS. 12 and 13 are diagrams explaining a flow of acquisition of depthinformation. FIG. 12 is a diagram illustrating a flow of data in afunctional block diagram of the image processing apparatus 14. FIG. 13is a flowchart presenting an image processing method using the imageprocessing apparatus 14.

First, the image acquisition unit 40 acquires an intraluminal image P(image acquisition step: step S101). Here, the intraluminal image P is aframe image 38 a constituting a moving image 38. Also, the scopeinformation acquisition unit 42 acquires scope information S (scopeinformation acquisition step: step S102). Next, the landmark recognitionunit 43 recognizes a landmark L in the intraluminal image P (landmarkrecognition step: step S103). Then, the scope information correctionunit 44 corrects the scope information S (scope information correctionstep: step S104). For example, the scope information correction unit 44acquires scope information T that is a change amount of the endoscopicscope 10 acquired based on the landmark L. Then, the depth informationacquisition unit 45 acquires a depth image I having depth information ofthe intraluminal image using the intraluminal image P and the scopeinformation T (depth information acquisition step: step S105). Then, thedisplay unit 16 displays geometric information relating to the lumen onthe display unit 16 based on the depth information (display controlprocessing step: step S106).

As described above, in the present embodiment, the landmark in theintraluminal image is recognized, and the scope information S iscorrected using the information relating to the recognized landmark.Accordingly, accurate depth information of an intraluminal image can beacquired based on the corrected accurate scope information T.

In the above-described embodiment, the example in which the scopeinformation T is acquired by correcting the scope information S acquiredby the scope information acquisition unit 42 using the informationrelating to the landmark has been described. However, the scopeinformation S acquired by the scope information acquisition unit 42 doesnot need to be corrected in some cases. That is, when correct scopeinformation T can be obtained by performing correction in the scopeinformation correction unit 44, it is preferable to correct the scopeinformation S. Such an embodiment will be described below.

Second Embodiment

Next, a second embodiment will be described. In the present embodiment,the scope information correction unit 44 corrects the scope informationS in accordance with recognition reliability of the landmark.

FIG. 14 is a flowchart presenting an image processing method accordingto the present embodiment.

First, the image acquisition unit 40 acquires an intraluminal image P(step S201). Also, the scope information acquisition unit 42 acquiresscope information S (step S202). Next, the landmark recognition unit 43recognizes a landmark L in the intraluminal image P (step S203).

Next, the landmark recognition unit 43 acquires recognition reliabilityof the recognized landmark (step S204). Here, the recognitionreliability of the landmark is acquired by various methods. For example,the landmark recognition unit 43 is constituted of a recognizer (learnedmodel) that has undergone machine learning, and a score obtained whenthe landmark is recognized can be used as the recognition reliability ofthe landmark.

Then, the scope information correction unit 44 determines whether therecognition reliability of the landmark is a threshold value or more(step S205). When the recognition reliability of the landmark is lessthan the threshold value, the scope information correction unit 44 doesnot correct the scope information S. In this case, the depth informationacquisition unit 45 acquires depth information using scope information Sof an actual measurement value that has not been corrected (step S207).In contrast, when the recognition reliability of the landmark is thethreshold value or more, the scope information correction unit 44corrects the scope information S (step S206). Then, the depthinformation acquisition unit 45 acquires depth information based on thecorrected scope information T (step S207). Then, the display unit 16displays geometric information relating to the lumen on the display unit16 based on the depth information (step S208).

As described above, when the landmark recognition unit 43 correctlyrecognizes the landmark, the scope information correction unit 44 cancorrectly acquire the change amount of the endoscopic scope 10 based onthe landmark. In contrast, when the landmark recognition unit 43 cannotcorrectly recognize the landmark, it may be difficult for the scopeinformation correction unit 44 to correctly acquire the change amount ofthe endoscopic scope 10 based on the landmark. Thus, in the presentembodiment, since the scope information S is corrected in accordancewith the recognition reliability of the landmark, accurate depthinformation can be acquired.

Third Embodiment

Next, a third embodiment will be described. In the present embodiment,scope information is corrected in accordance with a correction valueobtained by the scope information correction unit 44.

FIG. 15 is a flowchart presenting an image processing method accordingto the present embodiment.

First, the image acquisition unit 40 acquires an intraluminal image P(step S301). Also, the scope information acquisition unit 42 acquiresscope information S (step S302). Next, the landmark recognition unit 43recognizes a landmark L in the intraluminal image P (step S303). Next,the scope information correction unit 44 acquires a correction value(step S304).

The scope information correction unit 44 outputs the correction valueobtained from information relating to the landmark. For example, asdescribed in the first embodiment, when the information relating to thelandmark is depth information corresponding to the landmark L, the scopeinformation correction unit 44 acquires a change amount of theendoscopic scope 10 obtained based on the information relating to thelandmark, and acquires the difference between the scope information Tcorrected with the change amount and the scope information S before thecorrection as a correction value. Then, the scope information correctionunit 44 determines whether the correction value is a threshold value ormore (step S305). When the correction value is less than the thresholdvalue, the scope information correction unit 44 does not correct thescope information S, and the depth information acquisition unit 45acquires depth information (step S307). In contrast, when the correctionvalue is the threshold value or more, the scope information correctionunit 44 corrects the scope information S (step S306), and the depthinformation acquisition unit 45 acquires depth information (step S307).Then, the display unit 16 displays geometric information relating to thelumen on the display unit 16 based on the depth information (step S308).

As described above, when the correction value is the threshold value ormore, the difference between the scope information S that is the actualmeasurement value and the scope information T corrected with the changeamount of the endoscopic scope 10 acquired based on the landmark L islarge. Thus, the scope information correction unit 44 performscorrection on the scope information S. In contrast, when the correctionvalue is less than the threshold value, the difference between the scopeinformation S that is the actual measurement value and the scopeinformation T corrected with the change amount of the endoscopic scope10 acquired based on the landmark L is small. Thus, even though thescope information S is used as it is, the influence on the accuracy ofthe depth information is small, and hence the scope informationcorrection unit 44 does not correct the scope information S.Accordingly, in the present embodiment, since the scope information iscorrected in accordance with the correction value, accurate depthinformation can be efficiently acquired.

Others

In the above-described embodiment, the hardware structures of processingunits that execute various kinds of processing (for example, the imageacquisition unit 40, the scope information acquisition unit 42, thelandmark recognition unit 43, the scope information correction unit 44,the depth information acquisition unit 45, the display control unit 46,and the voice control unit 47) are various processors as describedbelow. The various processors include a central processing unit (CPU)that is a general-purpose processor that executes software (program) tofunction as various processing units; a programmable logic device (PLD)that is a processor whose circuit configuration can be changed aftermanufacture, such as a field programmable gate array (FPGA); a dedicatedelectric circuit that is a processor having a circuit configurationdesigned exclusively for executing specific processing, such as anapplication specific integrated circuit (ASIC); and the like.

One processing unit may be constituted of one of these variousprocessors, or may be constituted of two or more processors of the sametype or different types (for example, a plurality of FPGAs or acombination of a CPU and an FPGA). Alternatively, a plurality ofprocessing units may be constituted of one processor. As an example inwhich the plurality of processing units are constituted of oneprocessor, first, there is an embodiment in which one processor isconstituted of a combination of one or more CPUs and software, and theprocessor functions as the plurality of processing units, as typified bya computer such as a client or a server. Second, there is an embodimentof using a processor that implements the functions of the entire systemincluding a plurality of processing units by one integrated circuit (IC)chip, as typified by a system on chip (SoC) or the like. As describedabove, the various processing units are constituted using one or more ofthe above-described various processors as the hardware structures.

Further, more specifically, the hardware structures of these variousprocessors are an electric circuit (circuitry) obtained by combiningcircuit elements such as semiconductor elements.

Each of the above-described configurations and functions can beappropriately implemented by any hardware, software, or a combination ofboth. For example, the present invention can be applied to a program forcausing a computer to execute the above-described processing steps(processing procedures), a computer-readable recording medium(non-transitory recording medium) having such a program recordedtherein, or a computer capable of installing such a program.

Although the examples of the present invention have been describedabove, the present invention is not limited to the above-describedembodiments, and of course various modifications can be made withoutdeparting from the spirit of the present invention.

REFERENCE SIGNS LIST

-   -   9 endoscope system    -   10 endoscopic scope    -   11 light source device    -   12 endoscope processor device    -   13 display device    -   14 image processing apparatus    -   15 operating unit    -   16 display unit    -   17 speaker    -   20 insertion section    -   21 hand operation section    -   22 universal cord    -   24 rotation scale    -   25 soft part    -   26 bending part    -   27 tip part    -   28 imaging element    -   29 bending operation knob    -   30 air/water supply button    -   31 suction button    -   32 still image capturing instruction portion    -   33 treatment tool lead-in port    -   34 length scale    -   35 light guide    -   36 signal cable    -   37 a connector    -   37 b connector    -   40 image acquisition unit    -   41 CPU    -   42 scope information acquisition unit    -   43 landmark recognition unit    -   44 scope information correction unit    -   45 depth information acquisition unit    -   46 display control unit    -   47 voice control unit    -   48 memory

What is claimed is:
 1. An image processing apparatus comprising aprocessor, wherein the processor performs: image acquisition processingof acquiring a time-series intraluminal image captured by a scope of anendoscope; scope information acquisition processing of acquiring scopeinformation relating to a change of the scope; landmark recognitionprocessing of recognizing a landmark in the intraluminal image; scopeinformation correction processing of correcting the scope informationusing information relating to the landmark recognized in the landmarkrecognition processing; and depth information acquisition processing ofacquiring depth information of the intraluminal image using theintraluminal image and the scope information corrected in the scopeinformation correction processing.
 2. The image processing apparatusaccording to claim 1, wherein, with reference to a position of the scopeat a time T, the scope information acquisition processing acquires achange amount of an insertion length of the scope and change amountsrelating to bending and rotation of the scope at a time T+α.
 3. Theimage processing apparatus according to claim 1, wherein the scopeinformation acquisition processing acquires information relating to aninsertion length of the scope and bending and rotation of the scope froman operation of an operation section of the scope.
 4. The imageprocessing apparatus according to claim 1, wherein the landmarkrecognition processing recognizes a temporal change of a correspondencepoint of the landmark, and wherein the scope information correctionprocessing corrects the scope information using the temporal change ofthe correspondence point.
 5. The image processing apparatus according toclaim 1, wherein the landmark recognition processing outputs recognitionreliability of the recognized landmark, and wherein the scopeinformation correction processing determines whether to execute thecorrection of the scope information based on the recognitionreliability, and performs the correction based on a result of thedetermination.
 6. The image processing apparatus according to claim 1,wherein the scope information correction processing outputs a correctionvalue obtained from the information relating to the landmark, determineswhether to execute the correction based on the correction value, andperforms the correction based on a result of the determination.
 7. Theimage processing apparatus according to claim 1, wherein the processorperforms: display control processing of displaying geometric informationrelating to a lumen on a display unit based on the depth informationacquired in the depth information acquisition processing.
 8. The imageprocessing apparatus according to claim 7, wherein the geometricinformation is at least one of a shape of the lumen, a position of alesion, a position of the scope, or a position of a treatment tool. 9.An image processing method using an image processing apparatuscomprising a processor, the method, performed by the processor,comprising: an image acquisition step of acquiring a time-seriesintraluminal image captured by a scope of an endoscope; a scopeinformation acquisition step of acquiring scope information relating toa change of the scope; a landmark recognition step of recognizing alandmark in the intraluminal image; a scope information correction stepof correcting the scope information using information relating to thelandmark recognized in the landmark recognition step; and a depthinformation acquisition step of acquiring depth information of theintraluminal image using the intraluminal image and the scopeinformation corrected in the scope information correction step.
 10. Anon-transitory, computer-readable tangible recording medium havingrecorded therein a program for causing, when read by a computer, aprocessor of the computer to execute the image processing methodaccording to claim 9.